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ABSTRACT 


The  main  thrust  of  this  paper  will  be  centered  around 
the  issue  of  whether  the  Navy  should  restructure  the  Opera¬ 
tional  Test  and  Evaluation  Force  (OPTEVFOR)  along  the  lines 
of  the  Army's  and  Air  Force's  independent  test  agencies.  The 
paper  begins  v/ith  an  overview  of  the  weapon  system  acquisi¬ 
tion  process.  It  then  proceeds  to  demonstrate  how  test  and 
evaluation  (T&E)  fits  into  the  acquisition  process.  Finally, 
a  description  and  evaluation  of  the  each  of  the  service's 
independent  test  agencies  are  presented. 

The  researcher  suggests  that  the  perceived  problems  are 
not  caused  by  OPTEVFOR's  present  structural  arrangement,  but 
are  the  result  of  OPTEVFOR's  philosophy.  The  final  conclu¬ 
sion  is  that  OPTEVFOR  should  not  change  its  existing  struc¬ 
ture,  instead  it  needs  to  expand  upon  its  current  emphasis 
on  increased  communication. 
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INTRODUCTION 


I. 

Weapon  system  acquisition,  the  development  and  procure¬ 
ment  of  military  systems,  is  a  matter  of  widespread  and 
increasing  concern.  Government  Reports,  Congressional 
hearings,  as  well  as  articles  in  the  press,  have  suggested 
that  the  acquisition  process  is  not  working  as  well  as  it 
should.  A  few  examples  of  these  shortcomings  are:  that 
development  fails  to  produce  a  satisfactory  working  design, 
that  major  systems  incur  large  cost  overruns,  and  that  parts 
are  over  priced. 

Testing  of  new  weapon  systems  is  one  of  the  Department 
of  Defense’s  (DOD)  key  controls  in  the  process  of  acquiring 
these  complex  and  expensive  systems.  Adequate  test  and 
evaluation  (T&E)  of  these  weapon  systems  is  of  paramount 
importance  to  ensure  maximum  return  for  the  dollars  spent 
to  procure  the  systems. 

Significant  improvements  have  been  accomplished  in  the 
test  and  evaluation  portion  of  the  acquisition  process.  In 
earlier  years,  test  and  evaluation  was  often  accomplished 
when  or  where  time  allowed  and  usually  by  personnel  who  were 
not  trained  in  the  techniques  of  testing.  After  the  July 
1970  Blue  Ribbon  Defense  Panel  Reports,  test  and  evaluation 
policies  were  significantly  altered.  Separate  independent 
test  agencies  were  established  within  each  service.  These 
agencies,  although  organized  differently,  had  one  thing  in 
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common,  each  maintained  its  testing  independent  from  the 
developer,  reporting  directly  to  its  own  service 
headquarters.  These  independent  agencies  perform  what  is 
commonly  called  Operational  Test  and  Evaluation  (OT&E).  They 
check  to  see  if  and  how  well  the  system  will  function  in  an 
operational  environment. 

How  Operational  Test  and  Evaluation  is  being  implemented 
has  been  one  of  the  central  issues  identified  in  numerous 
committees,  commissions  and  reports  on  weaknesses  in  weapon 
system  acquisition.  The  issues  addressed  in  these  studies 
have  been  consistently  repeated.  A  report  done  in  February 
1977  by  the  Defense  Science  Board  identified  one 
significant  shortcoming  in  OT&E.  The  report  stated  that  it 
is  very  important  for  all  participants  in  the  acquisition 
process  to  participate  throughout  the  entire  acquisition 
process.  At  the  time  of  this  report  this  interaction  between 
the  developer  and  service  OT&E  organizations  did  not  exist. 
[Ref.  1:  p . 7 ] 

Directives  and  instructions  have  been  issued  which 
should  have  corrected  some  of  the  major  OT&E  problems.  These 
instructions  however,  have  not  been  effectively  implemented 
in  the  Navy,  and  this  has  resulted  in  the  Navy's  OT&E 
practices  remaining  essentially  unchanged  with  little 
interaction  between  the  developer  and  the  independent 
operational  test  agency.  This  course  of  action  has  been  one 
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of  the  reasons  for  an  increase  in  the  length  of  the 
acquisition  process,  which  in  turn  has  increased  the 
overall  costs  of  the  process  as  well. 

As  a  result  of  the  Navy's  reluctance  for  interaction,  a 
question  has  been  asked  about  the  present  structure  of  the 
Navy's  independent  operational  test  agency.  This  question  is 
should  the  Navy  restructure  its  Operational  Test  and 
Evaluation  Force  (OPTEVFOR)  along  the  lines  of  the  Army  and 
the  Air  Force?  Presently  the  Army  and  Air  Force  independent 
operational  test  agencies  act  as  a  manager,  allowing  the 
user  to  do  the  actual  operational  testing.  The  Navy's 
Operational  Test  and  Evaluation  Force  on  the  other  hand, 
acts  as  a  surrogate  for  the  fleet  (user)  and  does  all  the 
Operational  Test  and  Evaluation.  It  is  felt  by  some,  that 
by  restructuring,  the  user's  inputs  could  be  introduced  into 
the  process  on  an  earlier  and  continuous  basis.  The 
contention  is  that  the  restructuring  would  also  enormously 
reduce  the  size  and  cost  of  OPTEVFOR  and  additionally  would 
provide  a  trained  first  fleet  unit  at  the  end  of  the 
Operational  Evaluation  (OPEVAL).  An  additional  potential 
benefit  espoused  would  be  an  earlier  Initial  Operational 
Capability  (IOC).  [Ref.  2] 

The  main  thrust  of  this  thesis  will  center  around  the 
issue  of  whether  the  Navy  should  restructure  OPTEVFOR  along 
the  lines  of  the  Army  and  Air  Force.  However,  before  this 


8 


issue  can  be  properly  addressed,  a  foundation  has  to  be 
laid.  This  foundation  will  consist  of  an  overview  of  the 
weapon  system  acquisition  process,  how  T&E  fits  into  the 
acquisition  process,  and  finally  a  description  and 
evaluation  of  each  of  the  service's  independent  test 
agencies . 
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II.  DEFENSE  ACQUISITION  PROCESS 


The  issue  of  whether  to  restructure  the  Navy’s  Opera¬ 
tional  Test  and  Evaluation  Force  can  not  be  properly 
addressed  without  a  basic  understanding  of  the  DoD’s  weapon 
system  acquisition  process  and  how  the  Service’s  T&E  program 
fits  into  the  overall  acquisition  scheme. 

The  present  method  of  managing  the  weapons  systems  ac¬ 
quisition  process  in  the  DoD,  emerged  as  a  result  of  a  study 
by  the  President’s  Blue  Ribbon  Defense  Panel  (BRDP)  done  in 
1970.  As  a  result  of  recommendations  of  the  BRDP,  the 
Department  of  Defense  Directive  5000.1  was  promulgated  and 
released  in  1971.  The  next  item  to  be  released  in  this 
evolution  was  the  issuance  of  Office  of  Management  and 
Budget  (0MB)  Circular  A-109,  which  was  released  in  1976.  The 
policies  and  guidelines  in  these  documents  form  the  basis 
for  all  the  subsequent  directives  and  instructions  regulat¬ 
ing  the  acquisition  of  weapon  systems.  These  directives  and 
instructions  enable  the  Secretary  of  Defense  (SECDEF),  who 
is  aided  by  the  Joint  Requirements  and  Management  Board 
(JRMB),  formerly  called  the  Defense  System  Acquisition 
Review  Council  (DSARC),  to  guide  and  control  the  development 
and  production  of  major  weapon  systems  through  a  series  of 
acquisition  phases,  milestone  reviews,  and  decision  points. 
The  process  is  illustrated  in  Figure  1  [Ref.  3 : p .  6]. 
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It  is  the  Secretary  of  Defense  who  makes  the  decision  as 
to  which  new  systems  will  be  designated  a  major  system.  The 
decision  to  designate  any  system  as  major  is  based  upon: 

a.  Development  risk,  urgency  of  need,  or  other  items  of 
interest  to  the  SECDEF. 

b .  Joint  acquisition  of  a  system  by  the  DOD  and  repre¬ 
sentatives  of  another  nation,  or  by  two  or  more  DOD 
Components . 

c.  The  estimated  requirement  for  the  system's  research, 
development,  T&E,  procurement  (production)  and  opera¬ 
tion  and  support  resources.  A  Justification  for  Major 
System  New  Start  (JMSNS)  is  required  for  all  acquisi¬ 
tions  for  which  the  DOD  Component  estimates  cost  to 
exceed  $200  million  (FY80  dollars)  in  RDT&E  funds  or 
$1  Billion  (FY80  dollars)  in  procurement  (production) 
funds,  or  both. 

d.  Significant  congressional  interest .[ Ref . 4  :  pp.  5-6] 

It  should  be  noted  that  all  weapon  systems  acquisitions 

go  through  a  similar  process  to  the  one  outlined  here  for 
major  systems.  The  only  difference  is  that  decisions  are 
made  at  lower  levels  within  the  Services. 

There  are  four  distinct  phases  and  milestones  in  the 
system  acquisition  process.  The  actual  acquisition  of  a 
new  system  begins  with  either  a  decision  to  establish  new 
capabilities  in  response  to  a  technologically  feasible  op¬ 
portunity,  an  identified  deficiency  in  an  existing 
capability,  a  significant  opportunity  to  reduce  the  DoD  cost 
of  ownership,  or  in  response  to  a  change  in  the  national 
defense  policy  [Ref  4:  pp.3-4].  This  part  of  the  process  is 
called  Milestone  0. 
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By  preparing  a  Justification  for  Major  System  Mew  Start 
(JMSNS).  the  Services  document  the  need  for  the  new  system 
and  submit  it  into  the  Planning.  Programming .  and  Budget 
System  (PPBS)  process  along  with  the  Service's  Program 
Objective  Memorandum  (POM)  for  the  year  in  which  the  funds 
are  requested.  The  Secretary  of  Defense  may  than  sanction 
the  JMSNS  in  the  Program  Decision  Memorandum  (PDM).  This 
action,  designated  Milestone  0.  authorizes  the  Service  to 
initiate  the  program  when  funds  are  available  and  moves  the 
program  into  the  next  acquisition  phase  which  is  called 
Concept  Exploration  (CE). 

In  the  Concept  Exploration  phase,  several  contracts  are 
awarded  to  industry  to  identify  and  investigate  alternative 
system  design  concepts  that  will  satisfy  the  mission  need. 
At  the  conclusion  of  this  phase,  the  Program  Manager  (PM) 
makes  his/her  recommendation  for  those  concepts  which  have 
the  potential  for  further  development  and  evaluation  and 
should  be  carried  forward  into  the  next  phase  of  the  ac¬ 
quisition  process  which  is  called  the  Demonstration  and 
Validation  (DaV)  phase.  This  recommendation  is  documented  in 
the  System  Concept  Paper  (SCP). 

The  SCP  is  submitted  to  the  Joint  Requirements  and 
Management  Board  (JRMB)  and  then  forwarded,  if  approved,  to 
the  SECDEF.  The  SECDEF  uses  the  SCP  to  make  the  Milestone  I 
decision  .  The  authority  to  proceed  is  provided  in  the 
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Secretary  of  Defense  Decision  Memorandum  (SDDM).  Approval 
signifies  a  validation  of  the  requirement  and  is  authoriza¬ 
tion  to  proceed  with  the  D«V  phase  with  the  most  promising 
concepts  [Ref  4:  p.  4], 

An  additional  document  developed  during  the  CE  phase, 
is  the  Test  and  Evaluation  Master  Plan  (TEMP).  The  TEMP  is  a 
short,  concise  master  plan  for  T&E.  Its  purpose  is  to  iden¬ 
tify  all  required  T&E  resources  fl  facilitate  long-range  plan¬ 
ning,  programming,  and  budgeting  including  that  of  adequate 
numbers  of  test  hardware  items  and  specialized  major  range 
and  test  facilities^  ensure  accomplishment  of  adequate  T&Eti 
and  eliminiate  redundant  testing.  The  TEMP  forms  the  basic 
contract  between  the  Development  Agency  (DA)  which  is 
responsible  for  the  Development  Test  and  Evaluation  (DT&E) 
and  the  independent  test  agency,  who  is  responsible  for  the 
Operational  Test  and  Evaluation  (OT&E).  While  the  initial 
version  of  the  TEMP,  which  is  required  at  Milestone  I,  will 
lack  many  specifics,  through  a  continuous  revision  process 
the  TEMP  will  develop  the  necessary  detail.  [Ref.  3:p.  3-191 

A  positive  Milestone  I  decision  allows  the  system  to 
enter  the  Demonstration  and  Validation  phase.  The  purpose  of 
this  phase  is  to  further  develop  and  validate  the  alterna¬ 
tive  concepts  to  determine  which  concept(s)  should  progress 
into  the  Full  Scale  Development  (FSD)  phase  [Ref. 3:  p. 1-151. 
This  phase  involves  demonstration  of  the  technology  to  be 
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used  in  the  system  or  critical  subsystems  to  verify  perfor¬ 
mance  and  the  potential  suitability  of  the  concept  to  fill 
the  mission  need.  At  this  point,  scientific  and  technologi¬ 
cal  development  is  required  to  bring  the  concept  to  its 
fruition. 

It  is  during  the  D&V  phase  that  both  DT&E  and  OT&E  begin 
with  the  test  results  being  used  to  support  the  Milestone  II 
decision.  Identifying  critical  issues  and  areas  of  risk 
that  have  been  addressed  by  test  and  evaluation,  the  Deci¬ 
sion  Coordinating  Paper  ( DC P ) / I n t eg r a t ed  Program  Summary 
(IPS)  is  prepared  by  the  Program  Manager.  The  DCP/IPS 
provides  the  test  objectives  and  measures  of  effectiveness 
related  to  the  satisfaction  of  mission  need  and  resource 
requirements  that  apply  to  the  test  activity.  The  PM  submits 
the  document  through  the  JR MB  to  the  SECDEF  for  the 
Milestone  II  decision.  SECDEF  approval  authorizes  the 
program  to  proceed  into  the  Full  Scale  Development  phase 
( FSD ) . 

The  purpose  of  the  FSD  phase  is  to  produce  a  fully 
tested,  documented,  and  production-engi neered  design  of  the 
selected  concept  [Ref.  3:  p. 1—15).  This  phase  is  divided 
into  three  subphases:  engineering,  prototype,  and  pilot- 
production/transition  to  production.  It  is  during  this  part 
of  the  acquisition  cycle  that  two  major  evaluations  take 
place.  The  first  is  the  technical  evaluation  (TECHEVAL),  the 
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purpose  of  which  is  to  identify  technical  deficiencies  and 
determine  whether  the  design  meets  technical  specifications 
and  requirements  [Ref.  5:  p.2].  This  evaluation  is  a  very 
important  facet  of  development  testing,  for  if  the  system 
passes  the  TECHEVAL  it  goes  on  to  the  next  major  evaluation, 
which  is  called  Operational  Evaluation  (OPEVAL).  Given  the 
data  from  the  TECHEVAL,  OPEVAL  proceeds  to  test  the  system 
for  demonstration  of  operational  effectiveness  and  opera¬ 
tional  suitability,  in  addition  to  verification  of  fixes  for 
problems  discovered  in  the  TECHEVAL  [Ref.  5:  p.  4],  These 
formal  evaluations  are  performed  in  order  to  certify  readi¬ 
ness  for  the  production  phase. 

The  final  milestone  decision  point,  Milestone  III,  is 
to  decide  whether  or  not  the  system  is  to  enter  into  produc¬ 
tion  and  deployment.  The  PM  documents  the  results  of  the 
FSD  phase,  including  testing,  and  plans  for  future  testing 
are  assessed  again.  The  PM  then  sends  his  recommendations  in 
an  updated  DCP/IPS  through  the  JRMB  to  the  SECDEF  for  his 
decision.  Normally,  however,  the  Secretary  of  Defense 
delegates  the  Milestone  III  decision  to  the  Secretary  of  the 
branch  of  the  Service  that  is  responsible  for  the  system. 
For  less  than  major  systems,  this  decision  in  some  cases  is 
delegated  to  the  lowest  level  in  the  organization  at  which 
overall  view  of  the  program  rests.  This  is  the  point  were 
test  results  play  their  most  critical  role  because  they 
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assist  the  decision  maker  in  his  decision  wh ether  to  approve 
or  deny  production. 

An  affirmative  Milestone  III  decision  allows  the 
system  to  enter  the  Production  and  Deployment  Phase.  One  or 
more  contractors  are  awarded  a  production  contract  for 
either  low- rate  or  full  rate  production.  Even  after  the 
system  is  being  produced,  testing  does  not  stop.  The  system 
is  assessed  in  new  environments,  in  different  platform 
applications,  in  new  tactical  applications,  or  against  new 
threats . 

Besides  the  aforementioned  fixed  milestones,  additional 
program  reviews  can  be  held  at  any  point  in  the  program  when 
major  problems  arise.  Most  of  the  information  at  the  reviews 
comes  from  the  test  and  evaluation  results  and  is  used  to 
assess  the  programs  progress  and  its  likely  success 
[Ref  6 : p .  83],  Depending  on  the  findings,  a  program  may 
advance  to  the  next  phase,  be  canceled  or  just  held  at  its 
present  phase. 

In  summation,  the  acquisition  process  that  is  presented 
here  is  a  very  simplistic  overview.  This  process,  though 
conceptually  simple,  in  fact  is  extremely  detailed  in  its 
requ i remen ts .  Test  and  evaluation  is  an  essential  player  in 
the  Department  of  Defense's  acquisition  process,  in  that 
the  T&E  results  help  to  determine  the  future  outcome  of  a 
program . 
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III.  WHAT  IS  TEST  AND  EVALUATION  ? 


Test  and  evaluation  is  a  fundamental  and  integral  part 
of  the  acquisition  process.  Although  the  terms  "test"  and 
"evaluation"  are  used  jointly,  they  do  represent  two  dis¬ 
tinct  and  unique  functions.  One  definition  of  TEST  is  11  a 
critical  examination,  observation  or  evaluation;  a  trial  or 
group  of  trials”  [Ref.  7  :  p .  2].  Evaluation,  on  the  other 
hand  concerns  the  review  and  analysis  of  data  produced 
during  testing  or  use  of  the  item. 

The  Department  of  Defense  is  generally  concerned  with 
three  types  of  test  and  evaluation:  Development  Test  and 
Evaluation  (DT&E),  Operational  Test  and  Evaluation  (OT&E) , 
and  Production  Acceptance  Test  and  Evaluation  (PAT&E) 
[Ref. 8:  p.2].  Figure  2  is  a  summary  overview  of  the  acquisi¬ 
tion  process,  and  it  illustrates  where  these  three  types 
of  test  and  evaluation  fit  into  the  acquisition  process 
[Ref. 3:  p.  1-18]. 

A.  DEVELOPMENT  TEST  AND  EVALUATION  (DT&E) 

DT&E  is  that  test  and  evaluation  conducted  to 
demonstrate  that  the  engineering  design  and  development 
process  is  complete,  the  design  risks  have  been  minimized, 
the  system  will  meet  the  specifications,  and  to  estimate  the 
system's  military  utility  when  introduced  [Ref.  9:  p.  7-8]. 
Development  Test  and  Evaluation  is  planned,  conducted  and 
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monitored,  by  the  Developing  Agency  (DA)  of  the  Military 
Departments  or  other  Defense  Agencies,  with  the  results 
being  reported  to  the  responsible  Military  Service  Chief. 

DT&E  is  required  for  all  acquisition  programs.  It  can 
be  broken  down  into  three  major  phases:  DT-I,  DT-II,  and 
DT-III.  The  specific  objectives  of  each  phase  are  developed 
by  the  DA  and  are  published  in  the  Test  and  Evaluation 
Master  Plan  (TEMP). 

DT-I  is  conducted  during  the  Demonstration  and  Valida¬ 
tion  Phase  to  support  the  Milestone  II  decision.  Its  prin¬ 
cipal  purpose  is  to  demonstrate  that  all  technical  risks 
have  been  identified  and  that  solutions  to  the  risks  are  in 
hand  (Ref.  9:p.  7-81. 

Conducted  during  the  Full  Scale  Development  Phase,  DT-II 
is  used  to  support  the  production  and  deployment  (Milestone 
III)  decision.  DT-II  demonstrates  that  the  design  meets  its 
specifications  in  performance,  compatibility,  suppor- 
tability,  survivability,  safety,  and  human  factors  (Ref.  9: 
p.  7-9]. 

The  Technical  Evaluation  (TECHEVAL)  is  the  final  phase 
of  DT-II.  It  is  conducted  to  determine  whether  or  not  a 
system  is  functioning  in  a  technically  acceptable  manner, 
whether  it  meets  technical  and  design  performance  specifica¬ 
tions,  and  to  determine  if  it  is  ready  for  the  Operational 
Evaluation  (OPEVAL).  The  Developing  Agency  (DA)  has  the 
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primary  responsibility  for  planning  the  test  program  and  oo- 
taining  their  results.  After  the  TECHEVAL ,  the  DA  certifies 
that  the  system  is  ready  for  the  Operational  Evaluation 
(OPEVAL).  However*  OPEVAL  may  not  commence  until  the 
Secretary  and  the  independent  test  agency  of  the  concerned 
branch  of  the  service  accepts  the  DA's  certi f ica tion  of 
readiness  for  OPEVAL.  [Ref.  9:  p.  7-9] 

The  final  phase  of  DT&E  is  DT-III.  It  is  conducted  after 
the  production  and  deployment  decision  (Milestone  III).  DT-- 
III  is  used  to  verify  the  correction  of  design  deficiencies 
found  during  TECHEVAL  or  OPEVAL*  and  that  any  improvements 
that  were  made  to  the  system  are  effective. 

3.  OPERATIONAL  TEST  AND  EVALUATION  (OT&E) 

Operational  Test  and  Evaluation  is  the  test  and  evalua¬ 
tion  conducted  by  an  independent  agency  to  determine  the 
prospective  system's  military  utility*  operational  effec¬ 
tiveness  (including  compatibility,  interoperability, 
reliability*  and  maintainability),  and  logistic  and  training 
requirements  [Ref.  9:  p.  7-9].  Since  OT&E  is  conducted  under 
as  near  to  realistic  operating  conditions  as  possible, 
needed  modifications  to  the  system  do  become  apparent  during 
this  process.  The  operating  personnel,  as  well  as  required 
support  facilities,  are  typical  of  those  expected  to  operate 
and  maintain  the  system  when  it  is  deployed. 
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OT&S  can  be  broken  down  into  two  categories:  Initial 
Operational  Test  and  Evaluation  (IOT&E),  which  is  all  OT&E 
conducted  prior  to  the  production  and  deployment  decision, 
and  Follow-on  Operational  Test  and  Evaluation  (FOT&E),  which 
is  all  OT&E  which  is  conducted  after  the  production  and 
deployment  decision.  OT&E  can  also  be  divided  into  four 
phases  (two  in  IOT&E  and  two  in  FOT&E) . 

OT-I  is  the  first  phase  of  Initial  Operational  Test 
and  Evaluation  (IOT&E),  and  is  conducted  during  the 
Demonstration  and  Validation  Phase  to  support  the  full-scale 
development  decision  (Milestone  II).  Its  objectives  are  to 
provide  an  early  assessment  as  to  whether  or  not,  based  on 
the  system's  effectiveness,  the  project/program  should  con¬ 
tinue  on  its  development  [Ref.  9:  p.  7-9]. 

The  other  phase  of  IOT&E  is  called  OT-II,  and  is  con¬ 
ducted  during  the  Full-Scale  Development  Phase.  The  OPEVAL 
is  the  culmination  of  OT-II.  The  objectives  of  OT-II  are  the 
demonstration  of  the  achievement  of  program  objectives  for 
operational  effectiveness  and  of  operational  suitability. 

The  first  phase  of  Follow-on  Operational  Test  and 
Evaluation  (FOT&E)  is  called  OT-III,  and  is  conducted  after 
the  production  and  deployment  decision.  This  testing  is  done 
before  production  systems  are  available.  Normally  it  is 
done  using  the  same  preproduction  system  that  was  used  in 


22 


the  OPEVAL.  The  specific  objectives  of  OT-III  include:  test¬ 
ing  of  fixes  or  corrections  that  were  incorporated,  comple¬ 
tion  of  any  deferred  or  incomplete  Initial  Operational  Test 
and  Evaluation  (IOT&E).  and  evaluation  of  any  new  tactics 
development . 

OT-IV  is  the  second  phase  of  FOT&E,  and  it  is  conducted 
on  the  production  system.  One  of  the  initial  objectives  of 
OT-IV  is  the  demonstration  of  the  achievement  of  program  ob¬ 
jectives  for  production  system  operational  effectiveness  and 
operational  suitability,  especially  reliability,  main¬ 
tainability  and  logistic  suppor tabil i ty .  Additional  objec¬ 
tives  are  OT&E  of  the  system  in  new  environments,  in  new 
applications,  or  against  new  threats. 

C.  PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION  ( PAT&E ) 

Production  Acceptance  Test  and  Evaluation  is  conducted 
upon  production  items  to  ensure  that  the  items  procured  meet 
the  specifications  of  the  procurement  contract.  Successful 
completion  constitutes  the  recognition  that  the  item 
fulfills  the  requirement  for  which  it  was  produced.  PAT&E  is 
the  responsibility  of  the  Developing  Agency  and  begins  in 
the  transition  to  production  section  of  the  Full-Scale 
Development  Phase  and  continues  through  the  Production  and 
Deployment  Phase  [Ref.  9:  p.  7-9]. 

The  single  most  important  message  that  comes  out  of  the 
test  and  evaluation  evolution  is  that  any  new  acquisition 
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program  must  be  capable  of  meeting  or  exceeding  the  real 
world  goals  expected  of  it  in  the  actual  or  closely 
simulated  operational  environment  prior  to  any  major 
production  commitment.  The  information  obtained  by  T  &  E  is 
used  to  give  a  better  understanding  of  the  system's 
capabilities,  to  identify  shortcomings,  to  develop 
improvements,  and  to  assist  in  designing  the  system's 
future  replacements.  Additionally,  an  important  facet  of  the 
program  is  to  verify  the  ability  to  support  and  maintain  the 
system  and  to  develop  requirements  for  training  personnel  in 
the  operation  of  the  system. 
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IV.  THE  SERVICES T  INDEPENDENT  TEST  AGENCIES 

FOR  OPERATIONAL  TEST  AND  EVALUATION 


Having  established  a  base  from  which  to  work,  each  in¬ 
dividual  service  will  be  reviewed  to  see  how  they  are 
structured  to  handle  the  requirement  for  independent  test 
and  evaluation.  However,  before  this  is  undertaken,  the 
issue  of  what  prompted  the  drive  for  the  independent  test 
agencies  will  be  presented. 

During  the  period  prior  to  the  Presidents  Blue  Ribbon 
Defense  Panel  Report  (released  on  1  July  1970),  defense  sys¬ 
tem  planners  and  developers  received  little  attention  from 
Congress  on  the  methods  they  used  to  develop  and  test  the 
new  systems.  At  that  time,  most  of  the  controversy  sur¬ 
rounded  the  actual  need  for  the  weapon  system  and  the  cost 
of  placing  the  weapon  system  into  the  defense  inventory.  The 
concern  of  whether  the  weapon  system  could  in  fact  perform 
to  the  designated  specifications  or  was  actually  ready  for 
production  was  very  small  outside  the  Department  of  Defense. 
Most  of  this  lack  of  concern  can  be  attributed  to  a  general 
feeling  of  satisfaction  within  the  Congress  that  the  govern¬ 
ment  research,  development,  test,  and  evaluation  (RDT&E) 
side  had  been  very  successful  since  the  United  States  was 
winning  the  intercontinental  ballistic  missile  and  space 
race.  It  was  during  this  period  in  time  that  the  U.S.  placed 
a  man  on  the  Moon.  Another  factor  that  kept  the  Congress's 
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attention  was  the  war  in  Vietnam.  With  the  change  of  Presi¬ 
dents  and  a  change  in  attitudes  toward  economic  conditions 
and  domestic  issues,  the  Congress  intensified  its  scrutiny 
of  Defense  expenditures,  in  particular  the  area  of  major 
weapon  system  acquisition.  It  was  this  intensification  by 
the  Congress  that  brought  about  the  dramatic  change  in  the 
military  services'  test  and  evaluation  policies. 

The  President's  Blue  Ribbon  Defense  Panel  (BRDP)  was  one 
of  the  major  driving  forces  behind  the  one  hundred  and 
eighty  degree  change  in  attitudes.  In  the  BRDP  report  the 
Department  of  Defense's  operational  test  and  evaluation 
practices  were  specifically  addressed  as  a  significant 
problem  area.  The  following  excerpt  points  out  the  concern 
of  the  panel: 

There  has  been  an  increasing  desire,  particularly 
at  the  OSD  level,  to  use  data  from  OT&E  to  assist 
in  the  decision-making  process.  Unquestionably  it 
v/ould  be  extremely  useful  to  replace  or  support 
critical  assumptions  and  educated  guesses  with 
quantitative  data  obtained  from  realistic  and  relevant 
operational  testing. 

Unfortunately,  it  has  been  almost  impossible  to 
obtain  results  which  are  directly  applicable  to  the 
decision  or  useful  for  analysis.  Often,  test  data  do 
not  exist.  When  they  do,  they  frequently  are  derived 
from  tests  which  were  poorly  designed  or  conducted 
under  insufficiently  controlled  conditions  to  permit 
valid  comparisons.  It  is  especially  difficult  to 
obtain  test  data  in  time  to  assist  in  decision-making. 
Significant  changes  are  essential  if  OT&E  is  to  real¬ 
ize  its  potential  for  contributing  to  important  deci¬ 
sions,  particularly  where  the  tests  and  the  decisions 
must  cross  Service  lines.  [Ref.  10:  p.591 
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Secretary  Packard,  then  the  Deputy  Secretary  of  Defense, 
was  the  driving  force  within  the  Department  of  Defense  for 
changing  the  test  and  evaluations  policies.  He  personally 
sent  a  series  of  memoranda  to  the  Services  in  which  he 
requested  that  the  Services  arrange  for  OTAE  to  be  managed 
and  conducted  by  an  independent  agency  separate  from  the 
developing  command  and  which  reported  their  results  directly 
to  the  Chief  of  the  Service. 

The  formal  directive  came  in  the  form  of  two  Department 

of  Defense  Directives  (DODDs).  The  first  was  Department  of 

Defense  Directive  5000.1,  which  officially  tied  OT&E  with 

the  Joint  Requirements  and  Management  Eoard  (JRM3),  then 

called  DSARC,  Milestone  III  decision  point. 

Test  and  evaluation  shall  commence  as  early  as  pos¬ 
sible.  A  determination  of  operational  suitability, 
including  logistic  support  requirements,  will  be  made 
prior  to  large  scale  production  commitments,  making  use 
of  the  most  realistic  environment  possible  and  the  best 
representation  of  the  future  operational  system  avail¬ 
able.  The  results  of  this  operational  testing  will  be 
evaluated  and  presented  to  the  DSARC  at  the  time  of  the 
production  decision.  [Ref.  11:  para. Ill,  C .  1  ] 


The  second  directive  was  DODD  5000.3,  which  provided  ad¬ 
ditional  and  more  specific  test  and  evaluation  policy 
guidance  than  that  of  DODD  5000.1. 

In  each  DOD  component  there  will  be  one  major 
agency  separate  and  distinct  from  the  developing/ 
procuring  command  and  from  the  using  command  which  will 
be  responsible  for  0T&E  and  which  will: 

a.  Report  the  results  of  its  independent  test 
and  evaluation  directly  to  the  Military 
Service  Chief  or  Defense  Agency  Director. 
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b.  Recommend  directly  to  its  Military  Service 
Chief  or  Defense  Agency  Director  the  ac¬ 
complishment  of  adequate  OT&E. 

c.  Insure  that  the  OTdE  is  effectively  planned 
and  conducted  [Ref. 128:  para.  IV,  C . 1  ] . 

The  stage  had  now  been  set  with  the  official  Office  of 
the  Secretary  of  Defense  direction  for  the  formation  of  an 
independent  testing  agency  within  each  service. 

The  Blue  Ribbon  Defense  Panel  Report  should  be  con¬ 
sidered  the  catalyst,  which  resulted  in  a  major  change  in 
how  the  Services  viewed  their  test  and  evaluations  ac¬ 
tivities.  The  most  notable  change  came  in  the  area  of  OT&E. 
An  examination  of  how  each  branch  of  the  service  incor¬ 
porated  this  new  philosophy  follows. 

A.  ARMY 

During  the  period  1962-1971,  the  testing  of  systems  and 
equipment  under  development  and  in-process  of  acquisition  by 
the  Army  was  accomplished  almost  totally  within  the  Army 
Material  Command  ( AMC ) .  The  Army  Material  Command  assigned 
the  test  responsibilities  to  the  Commanding  General,  United 
States  Army  Test  and  Evaluation  Command  (TECOM). 

As  the  Army's  principal  material  testing  organization, 
the  Test  and  Evaluation  Command  was  assigned  the  basic  mis¬ 
sion  of  providing  the  decision  makers  with  unbiased  indepen¬ 
dent  appraisals  of  Army  material.  The  Army  chartered  the 
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TECOM  to  reduce  the  time  frame  between  design  and  production 
and  to  eliminate  duplication  of  effort  through  integrated 
testing  and  better  coordination. 

To  accomplish  what  the  charter  mandated.  TECOM  employed 
both  engineering  tests  and  service  tests.  Engineering  tests 
were  conducted  to  determine  if  the  system  really  met  the 
material  need.  Service  tests  examined  the  human  interface 
with  the  system.  These  tests  determined  the  equipment's 
suitability  for  the  Array's  use  and  for  release  to  produc¬ 
tion.  Actual  operational  units  were  employed  to  conduct 
tests  in  a  tactical  environment  and  in  general  possessed 
many  of  the  characteristics  of  present  0 T & E  practices. 
However,  these  tests  were  often  conducted  after  the  manage¬ 
ment  decision  to  produce  the  system  in  numbers  and  even 
after  the  deployment  for  Army  operational  use. 

Since  the  Test  and  Evaluation  Command  was  affiliated 
with  the  Array's  Material  Command  developers,  a  new  and  inde¬ 
pendent  command  was  necessary  in  order  to  meet  the  require¬ 
ments  of  the  Department  of  Defense  Directive.  As  a  result, 
the  Operational  Test  and  Evaluation  Agency  (OTEA)  was  estab¬ 
lished  on  25  September  1972  at  Fort  Belvoir,  Virginia. 
Through  their  charter  from  the  Secretary  of  the  Army,  OTEA 
was  given  the  management  responsibility  for  major  and  non¬ 
major  systems.  [Ref.  13:  pp.  C-1  -  C  —  3  3 
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The  Commanding  General  of  OTEA  reports  to  the  Chief  of 
Staff  of  the  Army.  The  mission  and  functions  of  OTEA  are  as¬ 
signed  by  Army  Regulation  (AR)  10-4,  ”U.S.  Army  Operational 

Test  and  Evaluation  Agency”.  The  mission  of  OTEA  ”is  to 
manage  all  user  testing,  operational  testing  (0T),  force 
development  testing  and  experimentation  (FDTE),  and  joint 
user  testing  directed  by  the  Office  of  the  Secretary  of 
Defense”  [Ref.  14:  p.  2].  The  specific  tasks  which  are  as¬ 
signed  to  OTEA  are  summarized  as: 

a.  Plan,  direct  and  evaluate  the  operational  testing  of 
all  major  and  selected  non-major  systems. 

b.  Coordinate  the  operational  testing  of  other  non¬ 
major  systems. 

c.  Manage  major  and  coordinate  non-major  FDTE. 

d.  Coordinate  Army  participation  in  the  planning  and 
execution  of  Joint  OT&E. 

e.  Provide  a  strong  focal  point  organization  to  keep 
the  Developing  Agency  (DA)  and  the  Office  of  the 
Secretary  of  Defense  (OSD)  fully  informed  on  the 
Army’s  OT&E  needs  and  accomplishments. 

The  original  manning  level  of  OTEA  was  set  at  120  person¬ 
nel,  which  included:  53  officers,  2  enlisted,  and  65 
civilians.  In  1975,  the  strength  was  increased  to  250  per¬ 
sonnel  with  125  officers,  20  enlisted,  and  105  civilians. 
Today,  its  strength  is  256  personnel  with  122  officers,  8 
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enlisted,  and  126  civilians  [Ref.  15].  Figures  3  shows 
OTEA's  relationship  within  the  Army's  Test  and  Evaluation 
organization.  Figure  4  is  the  organizational  chart  for  OTEA. 

When  the  Operational  Test  and  Evaluation  Agency  receives 
a  system  from  AMC  for  T&E,  it  either  appoints  its  own  test 
dir ec tor  and  team  or  tasks  the  Force  Command  (who  is  respon¬ 
sible  for  the  continental  U.S.  Army  Divisions)  along  with 
the  Material  Development  and  Readiness  Command  and/or  the 
Training  Doctrine  Command  to  perform  the  testing.  This  deci¬ 
sion  is  made  based  upon  the  size  and  importance  of  the 
program/system  to  be  tested.  OTEA's  role  when  it  retains  a 
system  for  testing  is  to  furnish  three  to  five  of  its  per¬ 
sonnel  to  fill  key  positions  on  the  team  such  as  the  Deputy 
Director  for  the  systems  OT .  the  Chief  Analyst,  the  Chief 
Data  Collector,  and  the  Chief  Controller.  TRADOC,  repre¬ 
senting  the  user,  provides  the  appropriate  assistants  in  the 
areas  of  conceptual  expertise.  The  remaining  positions  are 
filled  by  the  testing  command.  If  the  program/system  OTdE  is 
tasked  to  an  organization  other  than  OTEA,  OTEA  mainly 
evaluates  the  data  generated  by  the  testing  command.  OTEA 
is  currently  directing/managing  OTdE  for  62  major  and  250 
lesser  category  non-major  systems  (Ref.  16]. 
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Figure  3.  Army  Organization  for  Material  Acquisition  and  Test 
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OTEA  Organization  Chart 


B.  AIR  FORCE 


In  their  attempt  to  conform  to  the  DOD  requirement,  the 
Air  Force  initially  tried  to  achieve  independence  in  opera¬ 
tional  testing  by  placing  the  full  OT&E  responsibility  on 
the  user  Command.  Even  though  this  was  a  step  in  the  right 
direction,  the  Air  Force's  effort  fell  short  of  its  mark  and 
was  often  subject  to  criticism.  To  correct  this  shortcoming, 
the  Air  Force,  in  October  of  1974,  established  its  indepen¬ 
dent  test  agency  calling  it  the  Air  Force  Operational  Test 
and  Evaluation  Center  (AFOTEC)  at  Kirkland  Air  Force  Base, 
New  Mexico.  AFOTEC  is  a  separate  operating  agency  that 
reports  directly  to  the  Chief  of  Staff  of  the  Air  Force.  In 
addition  to  its  headquarters  at  Kirkland,  AFOTEC  has  four 
additional  permanently  established  detachments  at  Edwards 
Air  Force  Base,  California,  Eglin  Air  Force  Base,  Florida, 
Nellis  Air  Force  Base,  Nevada,  and  Kapaun  Air  Station,  West 
Germany.  Along  with  these  four  permanent  detachments  there 
are  additional  field  test  teams  at  various  designated  sites. 
[Ref.  17:  p. 1  1 

The  center's  headquarters  and  field  teams  include  opera¬ 
tional,  technical,  analytical  and  test  specialists.  The 
field  test  teams  conduct  their  test  at  the  selected  sites 
and  from  the  results  prepare  the  formal  reports.  These  field 
teams  are  composed  of  personnel  from  the  Air  Force  Test  and 
Evaluation  Center  along  with  personnel  from  the  operating 
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and  support  commands  who  will  eventually  receive  and  utilize 
the  systems  in  every  day  operation. 

The  Air  Force  Operational  Test  and  Evaluation  Center's 
stated  mission  is  "to  manage  the  Air  Force  operational  test 
and  evaluation  program  and  assess  the  military  utility  and 
operational  effectiveness  and  suitability  of  major  and 
designed  non-major  systems"  [Ref.  17:  p . 1 ] .  The  specific 
tasks  which  are  assigned  to  AFOTEC  are  summarized  as: 


a.  Design,  direct,  analyze,  evaluate  and  report  on 

OT&E  of  major  and  designed  non-major  Air  Force  sys 
terns . 

’o.  Designate  the  Deputy  Test  Director  and  provide  (with 
augmentation  from  major  commands)  the  OT&E  team  for 
combined  DT&E/IOTAE  programs  retained  by  AFOTEC. 

c.  Monitor  all  non-major  system  OT&E  conducted  by 
the  major  commands  (  e.g.  TAC,  SAC,  MAC)  through 
review  and  approval  of  tests  plans  and  review  of 
comments  on  the  test  reports. 

d.  Develop  policy  recommendations  for  Headquar ters , 

United  States  Air  Force  approval  and  subsequent  im¬ 
plementation  by  all  major  commands. 

e.  Act  as  spokesman  to  the  Joint  Requirement  and 
Management  Board  (JRMB)  on  matters  pertaining  to 
OT&E. 

f.  Support  Congressional  requests  for  OTAE  information 
on  weapon  systems  for  which  procurement  funds  are 
being  requested.  [Ref.  17:  p . 1  1 

Since  becoming  operational,  the  manning  of  AFOTEC  has 
increased  from  the  initial  authorization  of  208,  to  the 
present  strength  of  677.  This  manning  level  includes  477  of¬ 
ficers,  90  enlisted,  and  110  civilians.  AFOTEC  is 
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additionally  augmented  by  2500  personnel  from  the  previously 
mentioned  operating  and  support  commands  [Ref.  18]  The 
organization  of  AFOTEC  is  similar  to  that  of  the  Army's 
OTEA.  Figure  5  shows  the  overall  relationship  of  AFOT&E 
within  the  Air  Force,  and  Figure  6  is  the  organization  chart 
of  AFOTEC. 

Air  Force  Operational  Test  and  Evaluation  Center  is 
currently  managing  OT&E  for  65  major  and  designed  non-major 
programs.  In  addition,  the  center  is  presently  monitoring 
240  operational  test  efforts  being  done  by  other  major  USAF 
commands.  [Ref.  19] 


C.  NAVY 

The  Navy  had  an  easier  time  than  its  sister  Services  in 
coming  up  with  a  test  agency  that  was  independent  of  the 
developer,  since  the  Navy  already  had  one  in  existence.  This 
command  is  call  the  Operational  Test  and  Evaluation  Force 
( OPTEVFOR ) . 

OPTEVFOR  can  trace  its  origin  to  the  final  months  of 
World  War  II.  During  the  Okinawa  campaign,  the  surface 
forces  of  the  Navy  came  under  a  sustained  concentrated 
attack  by  Japanese  kamikaze  aircraft.  These  suicide  attacks 
took  a  great  toll  in  ships  and  personnel,  and  a  means  to 
deal  with  the  threat  was  urgently  required.  In  July  of  1945. 
the  Composite  Task  Force.  U.S.  Atlantic  Fleet  was  formed  and 
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Figure  5.  Air  Force  Organization  For  T&E 


AFOTEC  ORGANIZATION  STRUCTURE 


Figure  6. 


AFOTEC  Organization  Structure 


tasked  with  the  development  and  evaluation  of  methods  co 
combat  such  attacks.  This  was  the  origin  of  the  Navy's 
independent  test  agency.  [Ref.  13:  pp.D-1  -  D - 3  ] 

In  the  following  years,  changes  were  made  to  the  name  of 
the  command  and  to  the  assigned  mission  and  tasks.  These 
changes  resulted  in  a  wider  scope  of  responsibilities  and 
expanded  capabilities  in  test  and  evaluation.  Additionally, 
a  similarly  structured  command  was  formed  on  the  west  coast 
for  the  Pacific  Fleet. 

To  achieve  the  desired  independence  which  was  suggested 
in  the  Blue  Ribbon  Defense  Panel  report  and  required  in  the 
DOD  directives,  OPTEVFOR  had  to  make  a  shift  in  its 
emphasis.  This  change  in  mission  orientation  was 
accomplished  in  the  period  between  lace  1971  and  middle 
1972.  While  the  reorganization  did  not  involve  an 
appreciable  organizational  restructuring  or  an  increase  in 
manpower  allocation,  it  did  require  a  different  and  more 
expensive  mix  of  personnel.  The  Navy's  "hands-on" 
operational  testing  concept  required  operational  officers 
and  highly  skilled  noncommissioned  officers  that  were 
familiar  with  the  current  missions,  tactics,  policies,  and 
procedures.  [Ref.  13:  pp.  D-1  -  D  —  3 ] 

The  Operational  Test  and  Evaluation  Force  is  now  under 
the  direct  command  of  the  Chief  of  Naval  Operations  (CNO) 
for  the  conduct  of  OT&E.  The  organizational  relationship 
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that  OPTF.VFOR  has  'with  the  DOD/MAVY  can  be  seen  in  Figure  7. 
The  command  headquarters  for  OPTEVFOR  is  located  at  the 
Norfolk  (Virginia)  Naval  Base.  For  the  Pacific  area  of 
operations,  a  separate  staff  under  the  Deputy  COMOPTEVFOR  is 
located  at  the  Naval  Air  Station.  North  Island,  San  Diego, 
California.  The  function  of  the  Deputy  COMOPTEVFOR  is  to  act 
as  the  representive  of  OPTEVFOR  in  matters  in  the  Pacific 
area  and,  when  directed,  with  west  coast  operational 
commands  (e.g.  COMNAVAIRPAC ,  COHNAVSUFPAC ) . 

The  mission  of  OPTEVFOR  is  assigned  by  the  CNO  and  is 
"to  operationally  test  and  evaluate  specific  weapon  systems, 
ships,  aircraft,  and  equipments,  including  procedures  and 
tactics,  where  required;'  and,  when  directed  by  CNO,  assist 
development  agencies  in  the  accomplishment  of  necessary 
Developmental  Test  and  Evaluation"  [Ref.  20:  p.  1]. 
The  specific  tasks  which  are  also  assigned  to  COMOPTEVFOR  by 
the  CNO  are  summarized  as: 

a.  Function  as  an  independent  test  agency  for  OT&E 
under  the  command  of  the  CNO. 

b.  Serve  as  principal  advisor  to  the  CNO  for  all 
Department  of  the  Navy  OT&E  matters. 

c.  Present  results  of  OT&E  to  the  JRMB  at  Milestone 
III  (production  decision)  review  and  to  other  reviews 
as  directed  by  the  CNO. 

d.  Conduct  operational  tests  on  weapon  systems 
includng  ships  and  aircraft.  Evaluate  operational 
effectiveness,  suitability,  and  capability,  reporting 
the  results  to  the  CNO. 
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Figure  7. 


DoD/Navy  Organization  for  T&E 


e.  Assist  development  agencies  in  DT&E,  including  fleet 
support,  as  required,  reporting  the  results  of  such 
assists  and  an  assessment  of  the  system  tested. 

f.  Review  and  evaluate  the  T&E  planning  for  new  weapons 
systems  to  address  and  resolve  critical  issues  and 
report  the  findings  to  the  CMO  [Ref.  20:  p.  1]. 

The  current  authorized  personnel  strength  of  0PTEVF0R 
is  approximately  1400.  Its  Norfolk  Headquarters  has  268 
personnel,  including  119  officers.  119  enlisted,  and  30 
civilians  [Ref.  211.  This  represents  only  about  one-fifth  of 
the  authorized  positions  in  0PTEVF0R.  The  other  personnel 
are  spread  across  the  subordinate  command  ( DEPC0M0PTEVF0R ) . 
organic  aircraft  squadrons  (VX-1.VX-4.VX-5).  and  the 
Sunnyvale  detachment.  Figure  8  shows  the  current 
organizational  arrangement  of  OPTEVFOR. 

It  is  interesting  to  note  that  when  OPTEVFOR  does  OT&E 
on  a  ship  or  submarine  it  preforms  the  tests  in  a  very 
similiar  manner  to  that  of  the  Army  or  Air  Force.  OPTEVFOR 
usually  sends  a  detachment  of  their  people  to  the 
ship /submarine  to  direct  the  testing,  but  utilizes  that 
unit's  precommissioning  crew  instead  of  there  own  personnel 
for  conducting  operations.  In  the  case  of  an  aircraft  or 
related  hardware,  OPTEVFOR  uses  the  total  in-house  concept, 
using  only  their  own  personnel  to  test  the  item. 

Operational  Test  and  Evaluation  Force  is  presently 
managing  OT&E  for  92  major  and  non-major  systems  [Ref.  20]. 
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OPTEVFOR  ORGANIZATION  CHART 


Figure  8.  OPTEVFOR  Organization  Chart 


In  this  chapter  the  services'  independent  test  agencies 
have  been  reviewed.  Their  organizational  relationship  and 
methods  of  performing  OT&E  have  been  highlighted.  A  summary 
of  these  relationships  and  methods  are  provided  in  Appendix 

B. 
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V.  THE  ADVANTAGES  AND  DISADVANTAGES  OF  THE 
SERVICES'  INDEPENDENT  TEST  AGENCIES 

In  response  to  DoD  directives,  each  service  has  elected 
a  different  approach  in  their  organizational  structure  for 
performing  OT&E.  3oth  the  Army  and  the  Air  Force  chose  to 
participate  much  more  closely  with  the  Developing  Agency 
(DA).  Both  OTEA  and  AFOTEC  are  small  organizations  which 
plan,  direct,  and  control  the  operational  testing,  yet  they 
leave  the  actual  testing  to  the  user  command,  normally 
utilizing  the  unit  programmed  to  receive  the  first  opera¬ 
tional  system. 

The  Navy,  on  the  other  hand,  has  elected  to  use  the  to¬ 
tal  in-house  concept  in  the  case  of  aircraft  systems'  test¬ 
ing  and  evaluation.  OPTEVFOR  acts  as  a  surrogate  for  the 
user  by  utilizing  their  own  independent  test  facilities  and 
assets  to  conduct  tests  on  the  systems.  Additionally,  OPTEV¬ 
FOR  believes  that  the  involvement  with  the  DA  will  jeopard¬ 
ize  their  objectivity  with  the  system  [Ref.  22]. 

In  trying  to  answer  the  question  as  to  whether  OPTEVFOR 
needs  be  to  res  true  tured  ,  it  will  be  best  to  address  the 
advantages  and  disadvantages  of  each  service's  independent 
operational  test  agency. 

OPTEVFOR' s  strongest  feature  is  their  insistence  on 
total  independence  in  order  to  maintain  their  objectivity  in 
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the  T&E  process.  Currently,  OPTEVFOR  insists  that  in  order 
to  maintain  their  independence  or  objectivity,  interaction 
between  OPTEVFOR  and  the  DA  has  to  be  kept  to  a  minimum. 
Stated  another  way,  the  issue  is  independence  versus 
cooperation,  with  OPTEVFOR  placing  more  weight  on  the  side 
of  independence.  [Ref.  22] 

It  is  this  independence  that  is  considered  their 
strength  because  they  do  not  have  to  concern  themselves  with 
the  developer  trying  to  twist  their  arm  in  an  attempt  to  get 
a  favorable  decision.  Additionally,  by  making  themselves 
the  tester  and  not  using  the  operational  commands,  they  are 
able  to  address  the  system  with  more  objectivity  since  they 
do  not  have  a  vested  interest  in  the  system. 

OPTEVFOR's  insistence  on  independence  is  also  their  big¬ 
gest  disadvantage.  This  reluctance  has  been  the  focal  point 
in  numerous  studies,  such  as  the  the  President's  Blue  Rib¬ 
bon  Panel  Report  (BRDP)  in  1970.  The  BRDP  believed  that  it 
is  important  to  perform  OT&E  on  an  operationally  configured 
production  system.  However,  if  the  OT&E  process  only  com¬ 
mences  at  OPEVAL,  it  misses  most  of  the  opportunity  to  in¬ 
fluence  the  product  during  its  development  on  behalf  of  the 
operational  forces.  [Ref. 10:  p.  88]  The  Defense  Science 
Board  on  17  February  1977,  published  a  study  stating  that 
"interaction  among  development  test  and  evaluation  and 
operational  test  and  evaluation,  in  addition  to  the  close 
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contact  with  the  users,  pays  very  important  dividends  in 
terms  of  money,  time  and  operational  suitability"  [Ref.  1: 
p.  7].  Both  these  reports,  along  with  others,  indicate 
that  a  working  relationship  between  the  DA  and  the  indepen¬ 
dent  test  agency  is  critical  to  the  acquisition  process. 

Since  OPTEVFOR  does  not  actively  participate  in  the 
system,  this  tends  to  create  the  "gotcha"  syndrome 
[Ref.  253.  When  a  system  comes  from  TECHEVAL  to  OPEVAL. 
there  is  a  possibility  that  it  may  not  pass  due  to  a  change 
in  OPTEVFOR  test  criteria.  A  weapon  is  developed  to  meet  a 
stated  operational  requirement  (OR).  This  requirement  can  be 
affected  by  funding  reductions,  schedule  delays  and  contract 
disputes,  in  addition  to  changing  threats  .new  weapons  and 
new  tactics.  As  a  result,  the  DA  develops  a  system  that 
fully  meets  the  requirements  of  the  original  OR,  but  as  a 
result  of  the  aforementioned  ongoing  changes,  it  falls 
short  of  the  current  OT&E  criteria.  Since  OPTEVFOR  does 
not  interface  with  the  developing  agency,  it  is  extremely 
likely  that  the  DA  might  not  be  aware  of  these  changes.  The 
end  result  is  the  system  is  not  ready  for  OPEVAL  due  to 
test  criteria  changes.  It  is  at  this  point  that  the 
"gotcha"  takes  place. 

Another  study  done  in  the  summer  of  1984  by  the  Naval 
Research  Advisory  Committee  (NRAC)  further  illustrates  what 
the  results  can  be  when  there  is  little  or’  no  interface 
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between  OPTEVFOR  and  the  DA.  The  committee  reviewed  a 
random  sample  of  142  of  OPTEVFOR 's  OPEVALS  that  were 
conducted  between  1  975  and  1982.  The  results  of  the  study 
showed  that  20%  failed  OPEVAL  outright  and  47% 
conditionally  passed.  These  systems  were  assessed  for  their 
operational  effectiveness  and  suitability.  Of  those  that 
failed,  the  majority  of  them  failed  the  suitability  aspect 
of  the  test.  [Ref.  25]  The  question  which  arises  here  is. 
how  can  a  system  get  this  far  in  the  process  and  then  fail? 
This  is  like  having  a  student  advance  all  the  way  through 
the  education  system  and  then  give  this  student  a  literacy 
test  and  find  out  that  the  individual  is  unable  to  read  and 
write. 

In  the  author's  opinion,  the  number  of  failures  in 
OPEVAL  could  be  attributed  to  OPTEVFOR  insistence  on 
maintaining  their  independence.  With  increased 
communication.  OPTEVFOR  could  alert  the  development  agencies 
to  the  changes  in  the  suitability  requirements  much  earlier 
in  the  acquisition  process.  This  in  turn  would  reduce  the 
number  of  failures.  By  waiting  until  just  before  the  system 
is  ready  to  enter  production  to  communicate,  the  only 
recourse  if  a  discrepancy  is  discovered  is  to  fail  the 
system . 

On  the  other  side  of  the  spectrum  .  a  strong  point  of 
both  the  OTEA's  and  AFOTEC's  arrangement  is  their  increased 
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ability  to  communicate  with  the  DAs.  Instead  of  representing 
the  user  as  in  the  case  of  OPTEVFOR,  both  OTEA  and  AFOTEC 
believe  in  getting  the  user  involved  at  a  very  early  stage 
in  the  OT&E  process.  OTEA  uses  a  central  group  comprised  of 
their  personnel  and  the  user  to  test  a  system.  The  Air 
Force,  on  the  other  hand  obtains  the  personnel  needed  to 
test  the  system  from  the  user  and  support  commands  on  either 
a  Temporary  Additional  Duty  (TAD)  or  Permanent  Change  of 
Station  (PCS)  basis.  These  personnel  are  then  sent  to  AFOTEC 
to  support  the  program. 

OTEA  has  recently  implemented  a  new  plan  called 
Continuous  Conception  Evaluation  (C2E),  which  has  advanced 
the  idea  of  continuous  independent  evaluation  through  the 
program’s  acquisition  cycle.  Differing  from  OTEA's  previous 
methodology,  C2E  begins  at  a  much  earlier  point  in  the 
acquisition  process  and  follows  the  system  past  the 
production  phase  of  the  process.  The  data  generated  from 
■models,  simulations,  contractor/government  testing,  and 
operating  trials  is  now  being  collected  and  correlated  for 
future  use.  This  information  will  help  to  provide  trends, 
projections  and  inputs  for  existing  and  future  projects. 
[Ref.  23:  p . 6 ] .  Thru  C2E,  OTEA  is  now  in  essence  tracking  a 
major  system  from  inception  to  retirement  frora  inventory. 
Figure  9  shows  the  major  projects  that  are  currently  under 
C2E  [Ref.  23:  p.  10]. 
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AIR  DEFENSE 

1  ATM/JATM 

2  CHAPARRAL  RSS 

3  FAAOS 

FAAD  C2  I 
LOS  R  (PMS) 

LOS  F 
NLOS 

COMBINED  ARMS  AD 

4  MARK  XV  IFF 

5  MSAM 

6  PATRIOT  P3I 

7  STINGER  RMP 

ARTILLERY 

1  AFAS 

2  AFATOS 

3  ARMY  TACMS 

4  ETAS 

5  HIP 

6  MLRS  TGW 

7  PERSHING  II 

8  RPV 

9  SADARM 

10  XM119 


TACCS  LISTED  ELSEWHERE) 
INFANTRY 


1  AAWS  (L.  M.  HI 

2  BFV 

3  STINGRAY  (DEW) 
INTELLIGENCE 

1  ASAS 

2  IEW  UAV 

3  JSTARS 

COMBAT  SUPPORT 

1  ACE 

2  C17A 

3  HMMWV 

4  IRE 

5  M88  PIP 

6  TACCS 

7  XM40 


EMS  CURRENTLY  IN  C2E 

15  APRIL  1986 

ARMOR 

AVIATION 

1  ABRAMS 

1  AH64 

2  AFV 

2  AHIP 

COMMAND  h  CONTROL 

3  LHX 

1  ACCS 

SIGNAL 

2  MCS 

1  CSCE 

(AFATOS.  ASAS.  FAAD  C2  1  AND 

2  DGM 

TOTAL:  56 


3  I  S/A  AMPE 

4  MTCC 

5  MSE 

6  PLRS/JTIDS  HYBRID 

7  RECS 

8  SCOTT  MILSTAR 

9  SINCGARS  , 

10  SST 

11  AN/TRC  170 

12  AN/TTC  39A 

13  ULCS 

STRATEGIC 

1  SATKA  (AOA,  TIR,  SRHIT) 

2  KEW  IERIS,  HEOI,  LEOII 

3  DEW  (NPB.  FEL) 

4  SA/BM  IC3) 

5  WIS/AWIS 

6  NAVSTAR  GPS 


Figure  9.  Systems  Currently  in  C2E 


AFOTEC  is  similar  to  OTEA  in  that  it  establishes  a  close 
liaison  with  the  DA.  The  Air  Force's  new  emphasis  on  the 
operational  requirement  of  a  system  has  resulted  in  a  even 
closer  working  relationship  between  AFOTEC  and  the  DAs.  The 
DA  is  now  continually  kept  advised  of  any  changes  in  the 
system's  operational  requirements  by  AFOTEC.  The  obvious 
benefit  is  that  there  are  no  surprise  OR  changes  when  the 
system  enters  OPEVAL.  Additionally,  the  Air  Force,  in  an 
attempt  to  shorten  the  acquisition  process,  is  trying  to  use 
DT&E  to  help  verify  the  operational  performance  of  a  system. 
A  reduction  in  duplicative  testing  can  be  achieved  since 
some  of  these  tests  will  not  have  to  be  repeated  under  OT&E. 
Presently  being  used  on  the  B-1  bomber,  this  approach  is 
considered  a  significant  change  and  is  still  in  the  very 
early  stages  of  evolution  by  the  Air  Force.  [Ref.  24:  p.  5] 

This  close  working  relationship  of  OTEA  and  AFOTEC  with 
the  DA  and  user  commands  can  shorten  the  acquisition 
process.  3y  alerting  the  system  to  changes  and,  in  the  case 
of  the  Air  Force,  reducing  duplicative  efforts  in  testing 
the  total  time  required  to  field  a  system  is  reduced. 

The  underlying  reason  behind  the  establishment  of  the 
independent  operational  test  agency  (J.TA)  was  to  assure  the 
independence  in  the  testing  of  a  system.  However,  one  of 
the  biggest  disadvantages  that  can  result  from  too  close  a 
working  relationship  between  the  ITA  and  the  DA/user  command 
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is  a  loss  of  objectivity.  It  is  very  hard  to  maintain 
objectivity  when  so  much  time  has  been  spent  with  the  people 
developing  the  system.  Both  OTEA  and  AFOTEC  have  to  contend 
with  pressures  from  not  only  the  DA,  but  also  the  user, 
both  of  whom  are  trying  to  get  the  system  into  production. 
The  DA  wants  the  system  to  enter  production  because  this 
makes  him  look  good.  The  user  wants  the  hardware  since  it  is 
perceived  to  be  better  than  that  which  he  has  presently.  As 
a  result,  he  wants  the  system  to  do  well  in  testing  as  this 
would  expedite  the  fielding  of  the  new  system.  If  the  ITAs 
bends  to  their  wishes,  the  result  could  be  a  system  entering 
production  that  might  require  extensive  after-production 
modifications  to  make  it  function  properly.  Any  savings  in 
time  and  costs  which  were  gained  by  the  ITAs  interfacing  are 
now  removed  by  the  additional  time  and  costs  required  to  fix 
the  system's  shortcomings. 

In  summation,  this  chapter  has  illustrated  that 
OPTEVFOR's  strongest  point,  their  insistence  on 
independence,  is  also  their  weakest  feature.  In  their  effort 
to  insure  their  objectivity,  OPTEVFOR  has  missed  the 
opportunity  to  influence  the  system's  development  on  behalf 
of  the  operational  forces.  Additionally,  their  lack  of 
cooperation  can  add  to  the  length  of  the  acquisition 
process.  By  not  keeping  the  DAs  appraised  of  changes  in 
the  OR,  it  is  possible  that  a  system  might  have  to  go 


52 


bade  through  development  to  correct  the  shortcomings 
introduced  by  the  new  requirements.  On  the  other  hand. 
OTEA's  and  AFOTEC's  best  feature  is  their  strong 
relationship  with  the  DA.  This  relationship  can  correct 
OPTEVFOR's  most  prevalent  shortcomings,  however,  it  in 
itself  can  introduce  problems  if  carried  too  far,  resulting 
in  the  release  of  a  system  that  is  not  yet  ready  for 
production . 
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V 1 .  CONCLUSIONS/RECOMMENDATIONS 


Before  we  put  a  weapon  system  in  the  hands  of  our 
troops  we  should  ensure  that  it  performs  its  combat  mis¬ 
sion,  not  that  it  simply  meets  contract  specifications. 

.  .  .  American  service  personnel  have  a  right  to  ex¬ 
pect  that  the  government  will  provide  them  with  the 
necessary  tools  to  do  their  job.  [Ref.  26:  p.  23] 

This  statement  made  by  Representive  C.  E.  Bennett,  is  an 
excellent  summation  of  wha t  is  expected  from  OT&E  in  the  ac¬ 
quisition  process.  The  purpose  of  the  independent  test 
agencies  is  to  determine  whether  the  weapon  system  fulfills 
the  desired  function  in  an  operational  environment. 

In  the  introduction,  a  statement  from  the  Summer  1984 
NRAC  report  indicated  that  restructuring  would  enormously 
reduce  the  size  and  cost  of  OPTEVFOR.  In  addition,  the 
report  stated  that  the  user's  inputs  could  be  introduced 
into  the  process  on  an  earlier  and  continuous  basis  [Ref. 21. 
The  first  part  of  this  statement  is  appealing,  however  the 
preceding  analysis  of  operational  test  agencies  indicates 
that  there  is  strong  reason  to  believe  that  changes  in 
operating  policy,  not  the  actual  structure  of  OPTEVFOR, 
should  be  made. 

The  present  manning  levels  of  OTEA  and  AFOTEC  are  in  ac¬ 
tuality  about  the  same  or  even  greater  than  OPTEVFOR.  This 
analysis  of  the  manning  levels  is  based  on  counting  the  per¬ 
sonnel  that  OTEA  and  AFOTEC  utilize  from  other  commands  for 
support  of  their  testing.  By  comparing  these  numbers  against 
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OPTEVFOR's  total  complement  of  officers,  enlisted  and 
civilians,  it  becomes  evident  that  the  Army  and  Air  Force 
OT&E  manning  effort  is  not  less  than  the  Navy's.  For  ex¬ 
ample,  AFOTSC's  level  of  2500  supporting  personnel  is  al¬ 
most  double  that  of  0PTEVF0R  total  in-house  complement  of 
1300  personnel. 

Considering  the  costs  of  those  services  supplied  by  the 
other  supporting  commands,  it  appears  that  operating  costs 
of  each  ITA  are  comparable.  In  essence,  there  is  very 
little  difference  between  the  size  and  cost  of  each  ITA  if 
the  supporting  commands  for  OTEA  and  AFOTEC  are  included  in 
the  analysis.  The  actual  cost  of  each  ITA  and  its  supporting 
command  was  beyond  the  scope  of  the  research  for  this 
thesis,  however,  the  potential  value  of  this  information 
merits  further  study. 

The  statement  that  the  users  inputs  need  to  be  provided 
earlier  and  on  a  continuous  basis  can  be  viewed  as  a  result 
of  OPTEVFOR's  philosophy  and  not  its  existing  structure.  Un¬ 
til  recently,  0PTEVF0R  has  been  resistant  to  the  idea  of 
interacting  more  closely  with  the  DAs,  so  that  it  could 
maintain  objectivity.  The  importance  placed  by  0PTEVF0R  on 
maintaining  objectivity  has  changed  somewhat  with  the  new 
commanding  officer  (COMOPTEVFOR).  An  examination  of  the 
findings  of  the  Sumner  1934  NRAC  report  reveals  a  change  in 
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relationship  between  OPTEVFOR  and  the  DAs  [Ref.  25],  It  ap¬ 
pears  that  the  new  COMOPTEVFOR  endorses  increased  liaison 
with  the  DAs. 

In  the  past,  when  a  system  arrived  for  the  OPEVAL 
that  did  not  meet  the  test  criteria  it  would  immediately 
fail.  Now,  if  a  problem  is  discovered,  instead  of  failing 
the  system  outright,  CONOPTEVFOR  will  call  his  counterpart 
at  the  DAs  and  inform  him  that  the  system  is  not  doing  well. 
This  allows  the  DAs  to  remove  the  system  from  OPEVAL  and 
correct  the  deficiencies.  This  new  approach  not  only  keeps 
the  problems  in-house  and  away  from  the  public,  but  it  also 
fosters  greater  cooperation. 

This  cooperative  relationship  should  not  begin  just  at 
the  time  and  point  that  a  system  enters  OPEVAL.  It  is  not 
enough  to  just  advise  the  DAs  that  the  system  is  not  doing 
well  and  needs  some  correction.  The  Navy  needs  to  strongly 
encourage  the  early,  continuous,  and  positive  participation 
of  all  key  players  as  described  in  OPNAVINST  3960.1  0.  The 
difference  between  the  operational  test  requirements  and 
that  of  development  testing  mandates  closer  coordination 
between  the  Navy's  independent  test  agency  and  its  develop¬ 
ing  agencies.  These  organizations  need  early  and  clear 
definition  of  the  test  objectives,  criteria  and  the  stand¬ 
ards.  Once  these  conditions  can  be  agreed  upon,  they  must 
be  kept  current  throughout  the  program  to  minimize  the 
necessity  for  repeat  development  and  testing.  Failure  to  do 
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so  can  result  in  unnecessary  delays  and  higher  costs  in  the 
acquisition  of  fleet-ready  weapon  systems. 

Another  area  outside  the  scope  of  this  research  is  the 
cost  associated  with  having  to  send  a  system  back  through  a 
part  of  the  acquisition  process  because  it  fails  OPEVAL. 
There  are  undoubtedly  increased  costs,  however  the  magnitude 
of  this  problem  was  not  pursued.  This  subject  has  potential 
value  in  determining  the  real  efficiency  of  the  OT&E  effort 
and  merits  further  investigation. 

A  loss  of  independence  in  operational  test  is  not  being 
advocated  nor  is  inference  being  made  that  objectivity  is 
not  important.  The  real  key  to  effective  and  efficient  OT&E 
is  meaningful  communication  among  the  participants  in  the 
acquisition  process.  If  the  test  requirements  are 
coordinated  and  discussed  by  all  parties,  then  conflict  may 
be  reduced  to  minor  proportions  or  eliminated  altogether. 

OPTEVFOR  should  not  change  its  existing  structural 
arrangement,  since  it  been  has  demonstrated  that  this  was 
not  where  the  problem  exists.  Instead,  OPTEVFOR  needs  to 
expound  upon  the  new  movement  of  increased  communication 
with  the  DAs.  The  end  result  of  this  increased 
communication  will  be  a  vastly  improved  independent  test 
agency  that  may  ameliorate  the  potential  of  an  inadequate 
system  reaching  the  fleet.  The  Uavy  can  neither  afford 
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the  development  of  systems  that  do  not  meet  the  fleets 
needs,  nor  can  they  afford  the  publicity  that  results  in  a 
produced  system  that  fails  in  the  fleet. 
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APPENDIX  A:  GLOSSARY  OF  ACRONYMS  AND  ABBREV IAT JON'S 


ACAT 

Acquisition  Category 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation 
Center 

AFR 

Air  Force  Regulation 

AFSC 

Air  Force  Systems  Command 

AMC 

Army  Material  Command 

AR 

Army  Regulation 

ASARC 

Army  System  Acquisition  Review  Council 

ASD ( A&L ) 

Assistant  Secretary  of  Defense  (  Acquisition 
and  Logistics) 

ASM ( R&D ) 

Assistant  Secretary  of  the  Navy  for  Research 
and  Development 

BIS 

Board  of  Inspection  and  Survey 

BRDPR 

Blue  Ribbon  Defense  Panel  Report 

CD  EC 

Combat  Development  Experimentation  Command 

C  NO 

Chief  of  Navy  Operations 

C  OMOPTEVFOR 

Commander.  Operational  Test  and  Evaluation 
Force 

CSA 

Chief  of  Staff  Army 

CSAF 

Chief  of  Staff  Air  Force 

CTP 

Coordinated  Test  Program 

D&V 

Demonstration  and  Validation 

DA 

Developing  Agency 

DC  P 

Decision  Concept  Paper  (NAVY)  or  Development 
Concept  Paper  (ARMY) 
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DCP/ICP 

Decision  Coordinating  Paper/  Integrated 
Coordinating  Paper 

DCSRDA 

Deputy  Chief  of  Staff  Research  Development  and 
Acquisition 

DDT&E 

Deputy  Director  of  Defense  Research  and 
Engineering.  Test  and  Evaluation 

DDR&E 

Director  of  Defense  Research  and  Engineering 

DEPC0M0PTEVF0R 

Deputy  Commander  Operational  Test  and  Evaluation 
Force 

DoD 

Department  of  Defense 

DoDD 

Department  Of  Defense  Directive 

DOT&E 

Director.  Operational  Test  and  Evaluation 

DSARC 

Defense  Systems  Acquisition  Review  Council 

DT 

Development  Testing 

DT&E 

Development  Test  and  Evaluation 

FDTE 

Force  Development  Testing  and  Experimentation 

FOT&E 

Follow-on  Test  and  Evaluation 

FYTP 

Five  Year  Test  Plan 

GAO 

Government  Accounting  Office 

ILS 

Integrated  Logistics  Support 

IOC 

Initial  Operational  Capability 

IOT&E 

Initial  Operational  Test  and  Evaluation 

JMSNS 

Justification  of  Major  Systems  New  Start 

JRMB 

Joint  Requirement  and  Management  Board 

LOI 

Letter  of  Instructions 

MAC 

Military  Airlift  Command 

MAJCOM 

Major  Command 
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MASSTER 

Modern  Army  Selection  Systems  Test  Evaluation 
Review 

NAVAIR 

Naval  Air  Systems  Command 

OPEVAL 

Operational  Evaluation 

OPNAV 

Office  of  the  Chief  of  Navy  Operations 

OPNAVINST 

Office  of  Chief  of  Naval  Operations  Instruction 

OPTEVFOR 

Operational  Test  and  Evaluation  Force 

OR 

Operational  Requirement 

OSD 

Office  of  the  Secretary  of  Defense 

OT 

Operational  Testing 

OT&E 

Operational  Test  and  Evaluation 

OTEA 

Operational  Test  and  Evaluation  Agency 

PATdE 

Production  Acceptance  Test  and  Evaluation 

PCS 

Permanent  Change  of  Station 

PDA 

Program  Decision  Authority 

PDM 

Program  Decision  Memorandum 

PM 

Program  Manager 

PMP 

Program  Management  Plan 

POM 

Program  Objective  Memorandum 

PPBS 

Program,  Planning  and  3udget  System 

RAM 

Reliability,  Availability,  Maintainability 

RDT&E 

Research,  Development,  Test  and  Evaluation 

RDT&E 

Research.  Development,  Test,  and  Evaluation 

ROC 

Required  Operational  Capability 

SAC 

Strategic  Air  Command 
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SAM 

Systems  Acquisition  Management 

SCP 

System  Concept  Paper 

SDDM 

Secretary  of  Defense  Decision  Memorandum 

SECDEF 

Secretary  of  Defense 

SECNAVINST 

Secretary  of  the  Navy  Instruction 

SOR 

Specific  Operational  Requirement 

SPO 

System  Program  Office 

TAC 

Tactical  Air  Command 

TAD 

Temporary  Additional  Duty 

T&E 

Test  and  Evaluation 

TD 

Test  Directive 

TDP 

Technical  Development  Plan 

TECHEVAL 

Technical  Evaluation 

TECOM 

Test  and  Evaluation  Command 

TEMP 

Test  and  Evaluation  Master  Plan 

TRADOC 

Training  and  Doctrine  Command 

USACDC 

U.S.  Army  Combat  Developments  Command 

USD  (A) 

Under  Secretary  of  Defense  for  Acquisition 

VX 

Air  Test  and  Evaluation  Squadron 

62 


APPENDIX  B: 


ARMY/NAVY/  AIR  FORCE  COMPARISON 


US  AIR  FORCE 


1.  Designation  of 
major  field  agency 
responsible  for 
OT&E: 

Air  Force  Opera¬ 
tional  Test  and 
Evaluation  Center 
(AFOTEC)  HQ  AF  TEST 
AND  EVALUATION  CEN¬ 
TER 

Kirkland  Air 
Force  Base 
New  Mexico  87115 
Autovon:  244-0545 
Commercial:  (505) 
264-0545 

2.  Manning  Level: 

( authorized ) 

Military  567 
Civilian  1 10 

3.  Reporting  Chain: 

Reports  direct  to 
Chief  Staff  of  the 
Air  Force. 

4.  Principal 
directives/regula¬ 
tions  concerning 
OT&E: 

a .  AFR  80-14  "Test 

and  Evaluation" 

b.  AFR  23-36  "Air 

Force  Operational 
Test  and  Evaluation 
Center  (AFOTEC)" 


US  ARMY 


U  S  Army  Operational 
Test  and  Evaluation 
Agency  (0TEA)  CG,  HQ 
U  S  ARMY  OPERATIONAL 
TEST  and  EVALUATION 
AGENCY 

Falls  Church  VA. 
Autovon:  289-2228 
Commercial:  (703) 
756-1254 


Military  130 
Civilian  126 


Reports  direct  to 
Chief  Staff  of  the 
Army . 


a.  AR  70-10  "Test 
and  Evaluation 
During  Development 
and  Acquisition  of 
Material" 


US  NAVY 


Operational  Test  and 
Evaluation  Force 
(0PTEVF0R)  COM¬ 
MANDER,  OPERATIONAL 
TEST  and  EVALUATION 
FORCE 

Norfolk  VA.  23511 
Autovon:  564-5337 
Commercial:  (804) 
444-5337 


Military  238 
Civilian  30 


Reports  direct  to 
Chief  of  Navy  Opera¬ 
tions. 


a.  SECNAVINST  5000.1 
"System  Acquisition" 

b.  0PNAVINST  3960.10 
"Test  and  Eval¬ 
uation" 
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5.  Field  activities 
directly  under  OT&E 
Test  Agency: 

All  facilities  sup¬ 
port  and  personnel 
support  provided  by 
designed  MAJCOMS. 


6.  Comparison  of 
characteristics  of 
OT&E: 

a.  Development  of 
optimum  tactics, 
techniques,  proce¬ 
dures  and  concepts. 

b.  Evaluation  of 
reliability,  main¬ 
tainability,  and 
operational  effec¬ 
tiveness  and 
suitability. 

c.  Testing  under 
realistic  opera¬ 
tional  conditions. 

d.  Accomplish  IOT&E 
prior  to  first  major 
production  decision. 

e. FOT&E  subsequent 
to  receipt  of 
production  items. 

f.  IOT&E  will  not 
obviate  the  need  for 
FOT&E . 


b.  AR  10-4  "US  Army 
Operational  Test  and 
Evaluation  Agency" 


['one.  OTEA  con¬ 
tributes  deputy  test 
director  and  3  to  5 
key  personnel  to 
test  at  the  command 
conducting  test. 


a.  Representative 
user  troops 

b.  Realistic  opera¬ 
tional  environment 

c.  Provide  data  to 
estimate  military 
utility,  operational 
effectiveness,  and 
operational 
suitability  includ¬ 
ing  compatibility 
interoperability  , 
reliability  , 
availability  main¬ 
tainability  in¬ 
tegrated  logistics 
support  and  training 
requirements . 

d.  IOT&E  examines 
the  hardware  or  com¬ 
ponents  to  provide 
an  indication  of 
utility  and  worth  to 
the  user. 


c.  OP N AVI N ST 
5440.47  "Mission  and 
Function  of  Opera¬ 
tional  Test  and 
Evaluation  Force" 


a.  Three  Air,  Test 
and  Evaluation  Sqdns 
(VX-1 ,4, and  5) 

b.  Deputy  COMOPTEV- 
FOR  San  Diego 

c.  New  London  Test 
and  Evaluation 
Detachment 


a.  OT&E  par¬ 
ticipated  in  or  per¬ 
formed  by  opera¬ 
tional  personnel. 

b.  Focuses  on 
operational  effec- 
tiveness  and 
suitability  includ¬ 
ing  reliability, 
compatibility,  in- 
teroperability, 
maintainability  and 
supportability . 

c  .  Developmen  t  of 
optimum,  operational 
tac  tics . 

d.  IOT&E  a  c  - 
complished  prior  to 
JRMB  III  or  com¬ 
parable  CNO  major 
production  decision 
point . 
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g.FOT&E  focuses  on 
the  operational  and 
employment  aspects 
of  the  system. 

AFR  80-14 


7.  Authority  for  as¬ 
signment  of  0  T  &  E 
projects : 

a.  Chief  Staff  of 
the  Air  Force  (CSAF) 

b.  AFOTEC  may 
recommend  to  CSAF 
those  OT&E  projects 
which  it  or  MAJCOMS 
should  conduct. 


8.  Master  test  docu¬ 
ment  for  specific 
(major  or  selected 
non-major)  IOT&E 
program: 

Section  5  (Test  and 
Evaluation)  of 
Program  Management 
Plan  (PMP). 

9.  Views  on  combined 
DT&E/IOT&E : 

a. Operating  and 
supporting  commands 
participation  is 
mandatory  during 
development  testing. 


e.  Examination  of 
equipment  prior  to 
initial  production 
decision. 

f.  FOT&E  final  check 
prior  to  acceptance. 
AR  70-10 


Deputy  Chief  of 
Staff  Research 
Development  and  Ac¬ 
quisition  determines 
the  major  and  non¬ 
major  systems. 


Coordinated  Test 
Program  (CTP) 


a.  The  OT&E  process 
will  be  independent 
and  should  normally 
be  separate  from  the 
DT&E  process. 


e.  Assessment  of 
operational  effec- 
tiveness  and 
suitability . 

f.  FOT&E  conducted 
in  operational  en¬ 
vironment  by  opera¬ 
tional  personnel 
using  production 
systems . 

g.  Refine  tactical 
employment  doctrine 
and  requirements  for 
personnel  and  train¬ 
ing. 

OPNAVINST  3960.10 


a.  Chief  of  Naval 
Operations  (CN0) 
b  .  C  N  0  may 
authorize  C0M0PTEV- 
FOR  to  further  as¬ 
sign  projects  to 
other  commands  when 
advantages  are  ap¬ 
parent. 


Test  and  Evaluation 
Master  Plan  (TEMP) 


a.  0PTEVF0R  par¬ 
ticipates  in  the 
TECHEVAL  (  D  T  &  E  ) 
which  is  planned* 
conducted  by  the 
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b.  Operational  in¬ 
puts  will  be  util¬ 
ized  in  (test)  plan- 
ning  documents 
developed  by  the  im¬ 
plementing  Command. 

c.  To  maximum  pos¬ 
sible  degree.  1 0  T  «  E 
will  be  accomplished 
completed  by  operat¬ 
ing  personnel  of  the 
Operational/  Sup¬ 
porting  Commands 
using  realistic 
operational  environ¬ 
ment. 

d.  Where  adequate 
test  data  can  be 
secured  from  com¬ 
bined  DT&E/IOT&E, 
combined  tests  may 
be  employed.  Added 
considerations  for 
combined  testing  are 
costs  and  time  (AFR 
30-14) . 

e.  Separate  evalua¬ 
tions  required  from 
Operating/Supporting 
Commands  in  Produc¬ 
tion  Decisions. 


10.  Document  chain 
in  acquisition 
system : 

a.  ROC  (formally 
states  a  Required 
Operational 
Capabil ity ) 

b.  Action  Direc¬ 
tives  (early  PMD) 

c.  Program  Advocacy 
Documents 

d.  DCP  (OT&E  inputs 
provided  by  AFOTEC) 

e.  PMD  -  Initial 
Test  Directive  (OT&E 


However .  it  may  be 
combined  where 
separation  causes 
delay  involving  un¬ 
acceptable  risk  or 
unacceptable  ac¬ 
quisition  costs  (AR 
70-10) . 

b.  Testing  is 
usually  phased  as  DT 
I.  II.  and  III  and 
0T  I,  II.  and  III. 
Usually  0T  I  is  run 
in  combination  with 
DT  I.  Army  attempts 
to  keep  0T  II  and 
III  and  DT  II  and 
III  separated. 

c.  Each  phase  of  0T 
testing  results  in  a 
separate  evaluation 
by  OTEA,  timed  to 
provide  evaluation 
to  to  the  decision 
body  (JRMB)  at  deci¬ 
sion  milestones. 

d.  DT  II,  0T  II, 
and  DT  III  are  con- 
ducted  on  all 
development  items/ 
systems  as  a  mini¬ 
mum  . 


a.  ROC 

b.  Charter  for  spe¬ 
cial  task  force 

c.  OTEA  evaluation 
plan 

d.  Program  Advocacy 
Document 

e.  DCP  (OT&E  inputs 
provided  by  OTEA) 

f .  Final  Report  of 
Special  Task  Force 
(contains  plan  for 


Developing  Agency 
with  0PTEVF0R  in¬ 
puts.  0PTEVF0R 
provides  independent 
operational  assess¬ 
ment  to  the  CNO. 

b.  0PTEVF0R  is 
responsible  for 
planning,  conduct¬ 
ing,  reporting 
OPEVAL,  conducted 
after  TECHEVAL  and 
prior  to  Production 
Decision  milestone. 


a.  SOR  (formally 
states  a  Specific 
Operational 
Requirement ) 

b.  Technical 
Development  Plan 
(TPD)-  (program  ad¬ 
vocacy  document  in¬ 
cluding  OT&E  inputs 
by  0PTEVF0R ) 

c.  0PTEVF0R  evalua¬ 
tion  plan 
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inputs  provided  by 
AFOTEC) 

f.  Test  objectives 
annex  to  the  PHD 

g.  HO.  USAF  Test 
Directive  (Drafted 
by  AFOTEC) 
h  .  AFOTEC 
Commander's  Estimate 

i.  AFOTEC  Test  Plan 

j.  Program  Manage¬ 

ment  Plan  (contains 
D  T  &  E  ,  0  T  &  E  tests 

plans ) 

k.  Interim  Test 
Reports  (interim 
evaluation ) 

l.  Final  Test 
Report 

m  .  AFOTEC 
Commander's  Summary 
(Final  independent 
ops  evaluation) 


11.  Commands  con¬ 
ducting  OT : 

a.  AFOTEC-  (all 
major  and  selected 
nonmajor  systems) 

b.  Designed  AF 
MAJCOMS-  (nonmajor 
systems ) 


OTctE  orovided  by 
OTEA) 

g.  OTEA  or  TRADOC 
Outline  Test  Plans 

h.  Test  Plans 

i.  Development  Plan 
(contains  Coor¬ 
dinated  Test  Program 
of  all  DT  and  OT 
testing  of  the 
system ) 

j.  OT  I/DT  I  Test 
Reports  (independent 
operational  evalua¬ 
tion  by  OTEA) 

k.  OT  I I/DT  II  Test 
Reports  (independent 
operational  evalua¬ 
tion  by  OTEA) 

l.  OT  III/DT  III 
Test  Reports 
(independent  opera¬ 
tional  evaluation  by 
OTEA) 


a.  OTEA-  (all  major 
and  selected  non¬ 
major  systems) 

b.  Combat  Develop¬ 
ment  Experimentation 
Command  ( C  D  E  C  )  - 
(nonmajor  systems) 

c.  Modern  Army 
Selected  System 
Evaluation  and 
Review  (MASSTER)- 
(nonmajor  system) 

d.  Other  Major  Com¬ 
mands  (nonmajor 
systems ) 


d.  DC P  ( OT&E  inpu  L»  3 
provided  by 
OPTEVFOR ) 

e.  Test  and  Evalua¬ 
tion  Master  Plan 
(TEMP)-  (contains 
DT&E  and  OT&E  plans 
for  a  specific 
program 

f.  Project  M aster 
Plan  (  P  M  P  )  - 
(contains  the  TEMP) 

g.  TECHEVAL  Plan 
(the  developing 
agency's  plan  for 
DT&E ) 

h.  TECHEVAL  Report 

i.  OPTEVFOR's  inde- 
oendent  assessment 
of  TECHEVAL 

j.  OPEVAL  (OT&E) 
PLAN 

k.  OPEVAL  Report- 
(independent  opera¬ 
tional  evaluation  by 
OPTEVFOR) 


COMOPTEVFOR 
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12.  Commands  per¬ 
forming  evaluation 
of  OT&E  directed  by 
Service  HQ: 

The  command  conduct-  OTEA  performs  the 
ing  OT  performs  the  evaluation, 
evaluation  subject 
to  review,  approval, 
and  further  evalua¬ 
tion  by  AFOTEC. 


COMOPTEVFOR  performs 
the  evaluation. 
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